v0.72.0: restore redacted values in the form and depth each place needs - #334

What was wrong

Escaped originals. Outbound redaction finds values in the raw JSON body, so the ledger kept them with their JSON escapes. A PEM private key's newlines are \n there (two characters); a connection-string password can carry \\. When the model echoed the placeholder in a streamed answer, FrameRestorer put that raw form into a decoded string, so the client got a literal backslash-n instead of a newline. restore_json escaped the raw form again (double escaping), and the plugin bridge revealed raw forms into decoded text and failed to recognise the same key in decoded text (it was issued a second number).

Nested JSON. Whole-JSON restore (non-streamed answers, replays, WebSocket frames) escaped every value once, but a placeholder inside tool-call arguments sits in a string that is itself JSON text and needs escaping twice; the client's arguments did not parse.

Codex custom tool input. The Responses stream restorer had lanes for output_text and function_call_arguments, but not response.custom_tool_call_input.delta (freeform tools such as apply_patch). A placeholder split across those deltas reached the client unrestored.

WebSocket. Upstream frames were restored one by one on raw text, so a placeholder split across two delta frames was never restored.

Design

The ledger holds decoded originals

Each original is recorded decoded, whatever form it was found in:

Normalising at issue time (rather than tagging each entry with its form) also means the same value seen raw in the body and decoded by the plugin layer shares one placeholder number. Lookups that start from raw JSON text compare decoded values: stored-body masking and security-log contexts unescape the raw hit before looking it up, and security-log detail unescapes before placeholder_of. A value the client wrote with other escapes (\u00e9 from Python, \/ from PHP) is therefore stored as the placeholder the upstream received, not masked.

Every restore path writes the form and depth its destination needs

Structural whole-JSON restore. A small JSON reader (redact::nest) runs alongside the swap and tracks which string each placeholder sits in. A placeholder inside a string that is itself JSON text (Chat/Responses arguments, Anthropic partial_json, Converse toolUse.input, and only when that string starts with { or [, so a tool's own parameter named arguments holding a sentence stays text) gets the value escaped twice; elsewhere once. The reader nests recursively: a tool's own arguments field holding JSON text inside the arguments is one level deeper again, at any depth, and the streamed argument lanes (which read the decoded arguments with the same reader) agree with the whole-body restore. The JSON is not parsed and re-serialised: only the placeholders change, every other byte stays as it was (é, whitespace and key order included), bodies without placeholders are untouched, and the reader's state carries across chunks so the non-SSE byte stream needs no buffering. Text that is not JSON (it does not start with {, [ or ") is restored as before, escaped once.

FrameRestorer now leaves streamed fields to their lane: a whole placeholder inside a single arguments delta used to be restored by the frame-wide pass as plain text.

Custom tool call input

New Input(output_index) lane for response.custom_tool_call_input.delta, closed by response.custom_tool_call_input.done and response.output_item.done; a held tail is synthesized as a response.custom_tool_call_input.delta (same shape the dialect converter emits), not a function-call arguments delta. The lane restores the decoded value (freeform input is plain text).

WebSocket

Upstream text frames go through a per-connection FrameRestorer (Responses events), the same restorer as the SSE path: a placeholder split across delta frames is held per lane and restored; held tails are flushed as their lane's own events before the frame that closes the lane, or when the upstream goes away. When the ledger grows mid-connection (a later client frame carries a new value) the restorer switches to the new ledger and keeps every lane's held tail (FrameRestorer::rebook): the new ledger is a superset, so a held <<TW_SEC still joins RET_1>> from the next frame. Synthesized frames, including tails flushed when the upstream goes away, pass through model renaming, reply hooks and the tool wall like any other frame. Frames that are not JSON are restored whole with restore.

Public API (tw-guard)

• Added: replace::Form, replace::json_escaped, replace::apply_text, replace::restore_text, stream::Restorer::text, stream::Restorer::fresh_as, stream::Restorer::rebook, sse::FrameRestorer::rebook.
• Removed: replace::restore_json (folded into restore).
• Changed meaning, same signatures: Ledger::table/replacements/placeholder_of hold decoded originals; restore and Restorer::new escape by depth.
• The enterprise gateway uses flow::{look, replace, hits, hits_plain}, replace::{Ledger, restore}, sse::{FrameRestorer, Synth} and rules::* only; none of those signatures changed, and its restore / FrameRestorer callers get the fixes as they are.

Tests

• tw-guard
  • Ledger: PEM found in JSON is stored decoded, restore round-trips the body byte for byte, restore_text gives real newlines; escaped connection-string password; a literal backslash in decoded text survives; same value raw and decoded shares one number; trial sample with a typed \u0041 is not unescaped.
  • Nesting: the reader's levels for JSON-in-JSON-in-JSON (3 and 5 deep), Chat/Responses arguments, Converse toolUse.input, escaped quotes and \u0022 inside arguments, a tool parameter named arguments holding a sentence, Anthropic object input, non-JSON text, and the same levels however the text is cut; whole Chat/Responses/Converse/Anthropic answers restore to arguments that parse back to real newlines; a tool's own arguments field holding JSON parses back to real newlines at the innermost level, and streaming those arguments in 1/3/7/16-byte Chat deltas or a whole .done gives the same arguments as the whole-body restore; only placeholders move (\u00e9 and whitespace kept); ByteRestorer cut at every size from 1 to 39 bytes equals the whole restore.
  • SSE: PEM placeholder split in output_text deltas (Responses, Anthropic) gives real newlines; in function-call arguments (Responses deltas, .done, output_item.done, Chat, Anthropic partial_json) the arguments parse to real newlines; a whole placeholder in one arguments delta is escaped for the arguments; whole JSON answer escapes once; custom tool call input split across deltas is restored (deltas, .done, output_item.done) and a held tail is flushed as response.custom_tool_call_input.delta before .done / output_item.done / end of stream.
• tw-gateway
  • Plugin bridge recognises the body's PEM in decoded text as the same number and reveals real newlines; a decoded value with a literal backslash round-trips; stored body keeps escaped values as their placeholders, including values written with \u00e9 / \/ escapes serde would not produce; security-log detail names the placeholder of an escaped value and shows placeholders in contexts for such values.
  • End to end: a PEM echoed streamed and non-streamed comes back with real newlines; a WebSocket frame echoing a PEM placeholder comes back as the same valid JSON; a placeholder split across two upstream WebSocket frames comes back whole in both the text and the arguments lanes, and the .done frames carry it too; a half placeholder held while a later client frame grows the ledger is still joined with the next frame; a tail held when the upstream goes away reaches the client as a text delta.
• cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all -- --check pass.

🤖 Generated with Claude Code